{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div><img src=\"autoCV_logo.png\" width=\"220\" ALIGN=\"left\" border=\"20\"></div>\n",
    "<center><font color=\"4466ff\"> <h1>AutoCV Challenge Tutorial </h1> </font></center>\n",
    "\n",
    "ALL INFORMATION, SOFTWARE, DOCUMENTATION, AND DATA ARE PROVIDED \"AS-IS\".\n",
    "UNIVERSITE PARIS SUD, INRIA, CHALEARN, AND/OR OTHER ORGANIZERS\n",
    "OR CODE AUTHORS DISCLAIM ANY EXPRESSED OR IMPLIED WARRANTIES."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Before we start, you need to know that\n",
    "\n",
    "* We target applications of multi-label image classification. \n",
    "* Raw data are provided, in TFRecord format.  \n",
    "* We impose restrictions on training time and resources to push the state-of-the-art further.\n",
    "* This notebook uses sample data. Download larger datasets from the website of the challenge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os.path import join\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inges_dir = 'AutoDL_ingestion_program/'           # Ingestion program directory\n",
    "score_dir = 'AutoDL_scoring_program/'             # Scoring program directory\n",
    "model_dir = 'AutoDL_sample_code_submission/'      # Where model code is, called model.py\n",
    "baseline_dir = 'AutoDL_simple_baseline_models/'   # some baseline methods are implemented here\n",
    "\n",
    "output_dir = 'AutoDL_scoring_output'\n",
    "detailed_results_page = join(output_dir, 'detailed_results.html')\n",
    "\n",
    "from sys import path; \n",
    "path.append(model_dir); path.append(inges_dir); path.append(score_dir); path.append(baseline_dir);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Dataset Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>Let's start with a sample dataset miniciao, which can be found in <code>./AutoDL_sample_data/miniciao</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = 'AutoDL_sample_data'            # Change this directory and the dataset as needed\n",
    "data_name = 'miniciao'\n",
    "!ls $data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read train / test datasets\n",
    "from dataset import AutoDLDataset # The module 'dataset' is defined at AutoDL_ingestion_program/dataset.py\n",
    "D_train = AutoDLDataset(join(data_dir, data_name + '/' + data_name + '.data', \"train\"))\n",
    "D_test = AutoDLDataset(join(data_dir, data_name + '/' + data_name + '.data', \"test\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show important meta information about the dataset\n",
    "print (\"Dataset path: \", D_train.get_metadata().get_dataset_name())\n",
    "print (\"Image shape: \",  D_train.get_metadata().get_tensor_size(0))\n",
    "print (\"Dataset size: \", D_train.get_metadata().size())\n",
    "print (\"Output size: \",  D_train.get_metadata().get_output_size())\n",
    "print (\"Class labels: \", D_train.get_class_labels())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show sample images\n",
    "D_train.show_image(1);\n",
    "D_train.show_image(11);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>\n",
    "It should be noted that:\n",
    "    \n",
    "- in some datasets, the image shape is not fixed, i.e. some images are larger / smaller, when you get (-1, -1) as tensor size (image size) from meta information, it means that the image shapes are not identical in this dataset.\n",
    "- not all datasets have 3 channels\n",
    "- although this sample datset seems to have only one label for each image, it is not always true for other datsets.\n",
    "\n",
    "Thus, it depends on you to choose to adapt to different datasets. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Model, Prediction and Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy simple model\n",
    "model_simple = join(baseline_dir, 'linear', 'model.py') # choose one simple baseline model\n",
    "model_submit = join(model_dir, 'model.py') # submitted models must be called model.py\n",
    "!cp $model_simple $model_submit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set time budget and instanciate the model with dataset\n",
    "from model import Model\n",
    "time_budget=200\n",
    "M = Model(D_train.get_metadata())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train the model for a certain time\n",
    "M.train(D_train.get_dataset(), remaining_time_budget=time_budget)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get prediction by calling test method\n",
    "prediction = M.test(D_test.get_dataset(), remaining_time_budget=time_budget)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'> For each prediction made at a timestamp $t$, we compute for each (binary) class i the [ROC AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic): **$AUC_i$**, then normalize it (and average over all classes) by: \n",
    "\\begin{equation*}\n",
    "AUC = \\frac{1}{C} \\sum_{i=1}^C AUC_i, \\quad NAUC = 2 \\times AUC - 1,\n",
    "\\end{equation*}\n",
    "\n",
    "$NAUC$ is also called Gini index in some context. Then, the learning curve can be plot as a function of **$NAUC$** vs. time. Let's denote the learning curve as $s(t)$. Since $s(t)$ is defined as the $NAUC$ of the most recent prediction made before timestamp $t$, $s(t)$ is actually a **step function**.\n",
    "\n",
    "As this challenge aims to push forward the state-of-the-art in the **any-time learning** setting, we use a performance metric related to the whole learning curve (instead of only the last point). This metric is computed as follows.\n",
    "- In order to normalize time interval [0, T] to the [0, 1] interval, we perform a time transformation by\n",
    "  $$\\tilde{t}(t) = \\frac{\\log (1 + t / t_0)}{\\log( 1 + T / t_0)}$$\n",
    "  where $T$ is the time budget (of default value 1200 seconds = 20 minutes) and $t_0$ is a reference time amount (of default value 60 seconds).\n",
    "- Then we compute the area under learning curve using the formula\n",
    "  \\begin{equation*}\n",
    "    \\begin{aligned}\n",
    "    ALC &= \\int_0^1 s(t) d\\tilde{t}(t) \\\\\n",
    "    &= \\int_0^T s(t) \\tilde{t}'(t) dt \\\\\n",
    "    &= \\frac{1}{\\log (1 + T/t_0)} \\int_0^T \\frac{s(t)}{ t + t_0} dt \\\\\n",
    "    \\end{aligned} \n",
    "  \\end{equation*}\n",
    "  we see that $s(t)$ is weighted by $1/(t + t_0)$, giving a stronger importance to predictions made at the beginning of th learning curve.\n",
    "\n",
    "This gives the evaluation score used for one task. Later, when ALC score is computed for all tasks, the final score is obtained by the average rank (over all tasks). It should be emphasized that multi-class classification metrics are not being considered, i.e., each class is scored independently.\n",
    "\n",
    "Let's see in the following how the scores are computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate scores\n",
    "from score import autodl_auc, accuracy\n",
    "from libscores import read_array\n",
    "solution_file = join(data_dir, data_name + '/' + data_name + '.solution')\n",
    "solution = read_array(solution_file)\n",
    "\n",
    "acc = accuracy(solution, prediction) # note that accuracy is not evaluation metric in the challenge\n",
    "current_bac = autodl_auc(solution, prediction)\n",
    "# print('Number of test examples: %d \\n\\t\\t Solution \\t\\t\\t\\t\\t Prediction ' % len(solution))\n",
    "# [print(z) for z in zip(solution, prediction)]\n",
    "print (\"Classification Accuracy: \", acc)\n",
    "print (\"Normalized Area Under ROC Curve (NAUC) = {:.4f}.\".format(current_bac))\n",
    "print (\"ALC can be read from the result page as shown in the next part.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Test and Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>\n",
    "    It is important that you test your submission files before submitting them. All you have to do to make a submission is modify the file <code>model.py</code> in the <code>AutoDL_sample_code_submission/</code> directory, then run this test to make sure everything works fine. This is the actual program that will be run on the server to test your submission. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# run local test\n",
    "!python run_local_test.py -code_dir=./AutoDL_sample_code_submission -dataset_dir=AutoDL_sample_data/miniciao"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# result report\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML(detailed_results_page))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the learning curve we see that the predictions are only made at the beginning, then the training is stopped and no more predictions are made. This is due to several reasons:\n",
    "- the linear baseline by default only trains the model for 1 single epoch. This is specified by the attribute `self.num_epochs_we_want_to_train` in the class `Model`. (When this number of epochs of training is attained, the model will set `self.done_training` to `True` and ingestion program will stop the whole train/predict process and do final evaluation in scoring program);\n",
    "- the dataset `miniciao` is very small as it contains only 100 examples;\n",
    "- the neural network in this linear baseline method is very simple. Actually there is even no hidden layer in the neural network. \n",
    "All these factors together make the training (and testing/predicting) fast.\n",
    "\n",
    "You are invited to change the value of `self.num_epochs_we_want_to_train` in `model.py` and/or change the arguments (typically `code_dir` and `dataset_dir`) passed to `run_local_test.py` in a cell above to test different algorithms on different datasets, and hopefully get better performance than what we had. :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare a ZIP file ready for submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compress model to be submitted\n",
    "from data_io import zipdir\n",
    "submission_filename = 'mysubmission.zip'\n",
    "zipdir(submission_filename, model_dir)\n",
    "print(\"Submit this file: \" + submission_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>\n",
    "If you run the above cells successfully, congratulations! You are all set! In order to get better score on the challenge, you need to design your model carefully, that can learn better and faster on different datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>\n",
    "You don't need to write from scratch, but can instead follow our provided baseline model. Basically, you need to write three functions: (these functions can be found in <code>model.py</code>)\n",
    "\n",
    "- <code>preprocess_tensor_4d</code> (optional) for preprocessing data, e.g. resize, change gray images to RGB images\n",
    "- <code>input_function</code> (optional) for reading batchs\n",
    "- <code>model_fn</code> (mandatory) for defining your own models, CNN, ResNet, Inception, etc.\n",
    "\n",
    "For instructions on wrinting <code>model_fn</code>, you are invited to consult this page for reference: \n",
    "https://www.tensorflow.org/guide/custom_estimators#write_a_model_function\n",
    "\n",
    "Good luck!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
